Media content such as video content and audio content is commonly delivered to users in a digital form. If media content has a temporal aspect, and in particular is associated with a timeline which indicates how the media content is to be played out over time, such digital form is typically referred to as a media stream. Media streams may be delivered to a receiver of a user via a media distribution network. In particular, a media stream may be streamed to the receiver, which allows the receiver to begin play-out of the media stream before having received the entire media stream. However, the media stream may also delivered to the receiver of the user in a non-streaming manner, e.g., by being delivered to the receiver in the form of a file.
Examples of media streams include video streams such as camera-recorded or computer-rendered streams, audio streams such as microphone-recorded streams, timed text streams such as subtitle streams or social-media streams, timed events streams which show an advertisement image or perform an action at the receiver, and multimedia streams comprising different types of media streams.
Different media streams may be associated with a common play-out timeline in that there may be a temporal relationship between the media streams. In particular, media streams may be meant to be played out in a synchronized manner. This may be the case if the media streams concern different recordings of a same event, e.g., with a first media stream being a video stream and a second media stream being an audio stream, or the first media stream being a video stream representing a recording of the event from a first camera angle and the second media stream being also a video stream but representing a recording from a different camera angle.
Accordingly, there may be a need to enable play-out of different media streams in a synchronized manner. In addition, there may be a need to enable play-out of a same, similar or different media stream in a synchronized manner, e.g., across different receivers. The following are known examples where such media synchronization may be employed:                Social TV, in which synchronizing of a same or similar media streams across multiple TVs or other devices of multiple users is desired. The same or similar media streams may include technical variations of the same stream, but they may also include streams of a same event which nevertheless constitute different content, such as recordings from different camera angles of the same event.        Hybrid TV, in which synchronizing of multiple media streams, potentially coming via multiple routes, to a single TV is desired. Such multiple routes may include, e.g., Digital Video Broadcast (DVB), Internet Protocol (IP) multicast and IP unicast.        Companion screen, in which synchronizing a same or a different media stream between a TV and a companion screen, e.g., a tablet device, is desired. It is noted that the term ‘companion screen’ is also known as second screen.        
It is noted that the above examples primarily refer to television, but that similar examples exist for other types of devices or receivers. Moreover, in the above examples, some or all of the media streams may be real-time or quasi-real time media streams. Additionally or alternatively, some or all of the media streams may be cached, e.g., by the media distribution network, or recorded, e.g., by a Personal Video Recorder (PVR), or constitute so-termed catch-up TV or user trick-mode media streams.
Broadcasters may employ media synchronization to offer synchronized services, where users can combine media streams via multiple routes and/or on multiple devices in various broadcaster-offered combinations. Such synchronization may take place in a studio system of the broadcaster where all media streams come together. Also, value-added-service providers may employ media synchronization to offer value-added synchronization services. Examples of value-added media streams include media streams which provide commenting, subtitling, audio description or sign-language interpretation of another media stream, e.g., a broadcasted video stream.
A problem associated with media synchronization may be that, whereas different media streams may be synchronized at the broadcaster or third party, they may lose their synchronicity while being distributed to the end user. For example, media streams may delivered via different routes. These routes have different delays because of differences in distance (transmission speed), multiplexing and routing (buffers), cached and recorded delivery, signal processing (e.g. mixing, transcoding), etc. Similarly, in the case of a value-added-service provider, the value-added media stream may be delivered from the service provider, while the original broadcast stream may be delivered from the broadcaster, i.e., via a different route.
An article “Multimedia group and inter-stream synchronization techniques: A comparative study” by F. Boronat et al., Elsevier Information Systems, 34, 2009, pp. 108-131, provides a comprehensive overview of known inter-destination synchronization techniques, i.e., synchronization between different end-terminals.
Most of the referenced inter-destination synchronization techniques make use of timeline information which is comprised in the media stream. Here, the term ‘timeline information’ refers to metadata which enables a receiver to play-out media samples of the media stream according to a play-out timeline. For example, if the media stream is delivered via the Real-time Transport Protocol (RTP), i.e., constitutes a RTP media stream, timeline information such as a RTP timestamp may be used. Other examples of timeline information include the so-termed Program Clock Reference (PCR), Presentation Time Stamp (PTS) and Decode Time Stamp (DTS). By comparing such timeline information across different receivers, appropriate stream adjustments may be calculated. To enable such comparing across receivers, synchronized wall-clocks may be used on the receivers, or alternatively, high-quality signaling connections with little and/or known delays. For example, a delay of the play-out time of the media stream may be achieved by buffering the media stream at the receiver.
It has been recognized that media streams may lose their original timeline information in a media distribution network. Such media distribution networks are typically owned by cable operators, telecom operators and internet service providers, which are typically different from the broadcasters or value-added-service providers. In such media distribution networks, a media stream may be modified in head-ends in order to make the media stream suited for distribution over the networks. These networks may be, e.g., DVB based (DVB-T, DVB-C, DVB-H), IP based (RTP/RTCP, IP multicast, IP unicast). Different codecs (MPEG 2, H.264, HEVC), transport technologies (broadcast, streaming, progressive download, adaptive streaming), etc, may be used in the final delivery of the media stream. During one or more of these processes, the media stream may be stripped of its original metadata. As a result, the original timeline information may not be included in the modified media stream. Instead, new metadata may be included which may include new timeline information.
Examples of such modifying of a media stream include:                (Re-)multiplexing. Most multiplexers for MPEG TS (Transport Stream) generate new PCR/PTS/DTS values while multiplexing. Typically, however, only the absolute values of PCR/PTS/DTS are changed, whereas their relationship across the media streams is maintained. As such, audio-video lip sync may be preserved.        Transcoding and re-encoding. Here, all relationships between incoming and outgoing media samples may be lost. For example, in case of a video stream, frames of the video stream may become a different type, e.g. from I-frame to B-frame. In addition, if the frame-rate changes during transcoding and re-encoding, the one-to-one relationship between incoming and outgoing frames may be lost. Furthermore, the whole container of the media stream may be changed, e.g. from so-termed TS to ISOBFF, thereby changing also the type of timeline information.        Technical or business reasons. Media distribution networks may refuse to pass timeline information. For instance, a DVB Synchronized Auxiliary Data Packet Identifier (SAD-PID) may be stripped from a Transport Stream (TS) by default. This may be on purpose, e.g. for said business reasons, or the parameter may just not be supported by the current network, thereby constituting a technical reason.        
In addition to media streams being modified and thereby losing their original timeline information, such timeline information may also inherently differ between different media streams which are intended to be played synchronously but which are already initially, e.g., at a media stream origination, not provided with common timeline information. Also here there may be a need for media synchronization.
WO 2010106075 A1 describes a method and a system for inter-destination media synchronization of at least a first and a second (media) stream, wherein the second stream is the output stream of a stream modification unit using the first stream as an input stream. The method comprises the steps of: providing first arrival time information of a packet in the first stream arriving at a first synchronization point and second arrival time information of a packet in the second stream arriving at a second synchronization point; providing synchronization correlation information on the synchronicity relationship between said input stream and said output stream; and, calculating delay information on the basis of the first and second arrival time information and the synchronization correlation information.